When Online Learning Meets ODE: Learning without Forgetting on Variable Feature Space

Machine learning systems that built upon varying feature space are ubiquitous across the world. When set of practical or virtual features changes, online approach can adjust learned model accordingly rather than re-training from scratch and has been an attractive area research. Despite its importance, most studies for algorithms capable handling have no ensurance stationarity point convergence, while accuracy guaranteed methods still limited to some simple cases such as L_1 L_2 norms with square loss. To address this challenging problem, we develop efficient Dynamic Feature Learning System (DFLS) perform on unfixed more general statistical models demonstrate how DFLS opens up many new applications. We first achieve accurate & reliable feature-wise a broad class like logistic regression, spline interpolation, group Lasso Poisson regression. By utilizing DFLS, updated is theoretically same trained using entire space. Specifically, reparameterize feature-varying procedure devise corresponding ordinary differential equation (ODE) system compute optimal solutions status. Simulation reveal proposed substantially ease computational cost without forgetting.